On the Complexity of the Class of Regions Computable by a Two-Layered Perceptron

This work is concerned with the computational complexity of the recognition of \mbox{LP}_2, the class of regions of the Euclidian space that can be classified exactly by a two-layered perceptron. Several subclasses of \mbox{LP}_2 of particular interest are also considered. We show that the recognition problems of \mbox{LP}_2 and of other classes considered here are intractable, even in some favorable circumstances. We then identify special cases having polynomial time algorithms

On the Complexity of Recognizing Regions Computable by Two-Layered Perceptrons

This work is concerned with the computational complexity of the recognition of $ÞPtwo$, the class of regions of the Euclidian space that can be classified exactly by a two-layered perceptron. Some subclasses of $ÞPtwo$ of particular interest are also studied, such as the class of iterated differences of polyhedra, or the class of regions $V$ that can be classified by a two-layered perceptron with as only hidden units the ones associated to $(d-1)$-dimensional facets of $V$. In this paper, we show that the recognition problem for $ÞPtwo$ as well as most other subclasses considered here is \NPH\ in the most general case. We then identify special cases that admit polynomial time algorithms

On the Power of Democratic Networks

Linear Threshold Boolean units (LTU) are the basic processing components of artificial neural networks of Boolean activations. Quantization of their parameters is a central question in hardware implementation, when numerical technologies are used to store the configuration of the circuit. In the previous studies on the circuit complexity of feedforward neural networks, no differences had been made between a network with ``small'' integer weights and one composed of majority units (LTU with weights in {-1,0, 1}), since any connection of weight w (w integer) can be simulated by |w| connections of value Sgn(w). This paper will focus on the circuit complexity of democratic networks, i.e. circuits of majority units with at most one connection between each pair of units. The main results presented are the following: any Boolean function can be computed by a depth-3 non-degenerate democratic network and can be expressed as a linear threshold function of majorities; AT-LEAST-k and AT-MOST-k are computable by a depth-2, polynomial size democratic network; the smallest sizes of depth-2 circuits computing PARITY are identical for a democratic network and for a usual network; the VC of the class of the majority functions is n 1, i.e. equal to that of the class of any linear threshold functions

Bounds on the Degree of High Order Binary Perceptrons

High order perceptrons are often used in order to reduce the size of neural networks. The complexity of the architecture of a usual multilayer network is then turned into the complexity of the functions performed by each high order unit and in particular by the degree of their polynomials. The main result of this paper provides a bound on the degree of the polynomial of a high order perceptron, when the binary training data result from the encoding of an arrangement of hyperplanes in the Euclidian space. Such a situation occurs naturally in the case of a feedforward network with a single hidden layer of first order perceptrons and an output layer of high order perceptrons. In this case, the result says that the degree of the high order perceptrons can be bounded by the minimum of the number of inputs and the number of hidden units

Support Vector Machine for Multiclass Classification

Support vector machines (SVMs) are primarily designed for 2-class classification problems. Although in several papers it is mentioned that the combination of $K$ SVMs can be used to solve a $K$-class classification problem, such a procedure requires some care. In this paper, the scaling problem of different SVMs is highlighted. Various normalization methods are proposed to cope with this problem and their efficiencies are measured empirically. This simple way of using SVMs to learn a $K$-class classification problem consists in choosing the maximum applied to the outputs of $K$ SVMs solving a \textit{one-per-class} decomposition of the general problem. In the second part of this paper, more sophisticated techniques are suggested. On the one hand, a stacking of the $K$ SVMs with other classification techniques is proposed. On the other end, the \textit{one-per-class} decomposition scheme is replaced by more elaborated schemes based on error-correcting codes. An incremental algorithm for the elaboration of pertinent decomposition schemes is mentioned, which exploits the properties of SVMs for an efficient computation

DynaBoost: Combining Boosted Hypotheses in a Dynamic Way

We present an extension of Freund and Schapire's AdaBoost algorithm that allows an input-dependent combination of the base hypotheses. A separate weak learner is used for determining the input-dependent weights of each hypothesis. The error function minimized by these additional weak learners is a margin cost function that has also been shown to be minimized by AdaBoost. The weak learners used for dynamically combining the base hypotheses are simple perceptrons. We compare our dynamic combination model with AdaBoost on a range of binary and multi-class classification problems. It is shown that the dynamic approach significantly improves the results on most data sets when (rather weak) perceptron base hypotheses are used, while the difference in performance is small when the base hypotheses are MLPs

On the Decomposition of Polychotomies into Dichotomies

Many important classification problems are \emph{polychotomies}, \emph{i.e.} the data are organized into $K$ classes with $K>2$. Given an unknown function $F : Ømega \to \{1, \dots, K\}$ representing a polychotomy, an algorithm aimed at ``learning'' this polychotomy will produce an approximation of $F$, based on a set of pairs $\{(\mathbf{x}^p, F(\mathbf{x}^p))\}_{p=1}^P$. Although in the wide variety of learning tools, there exist some learning algorithms capable of handling polychotomies, many of the interesting tools were designed by nature for dichotomies ($K=2$). Therefore, many researchers are compelled to use techniques to decompose a polychotomy into a series of dichotomies and thus to apply their favorite algorithms to the resolution of a general problem. A decomposition method based on error-correcting codes has been lately proposed and shown to be very efficient. However, this decomposition is designed only on the basis of $K$ without taking the data into account. In this paper, we explore alternatives to this method, still based on the fruitful idea of error-correcting codes, but where the decomposition is inspired by the data at hand. The efficiency of this approach, both for the simplicity of the model and for the generalization, is illustrated by some numerical experiments

Constructive Training Methods for Feedforward Neural Networks with Binary Weights

Quantization of the parameters of a Perceptron is a central problem in hardware implementation of neural networks using a numerical technology. A neural model with each weight limited to a small integer range will require little surface of silicon. Moreover, according to Occam's razor principle, better generalization abilities can be expected from a simpler computational model. The price to pay for these benefits lies in the difficulty to train these kind of networks. This paper proposes essentially two new ideas for constructive training algorithms, and demonstrates their efficiency for the generation of feedforward networks composed of Boolean threshold gates with discrete weights. A proof of the convergence of these algorithms is given. Some numerical experiments have been carried out and the results are presented in terms of the size of the generated networks and of their generalization abilities

Combining Linear Dichomotizers to Construct Nonlinear Polychotomizers

A polychotomizer which assigns the input to one of $K, K \ge 3$, is constructed using a set of dichotomizers which assign the input to one of two classes. We propose techniques to construct a set of linear dichotomizers whose combined decision forms a nonlinear polychotomizer, to extract structure from data. One way is using error-correcting output codes (ECOC). We propose to incorporate soft weight sharing in training a multilayer perceptron (MLP) to force the second layer weights to a bimodal distribution to be able to interpret them as the decomposition matrix of classes in terms of dichotomizers. This technique can also be used to finetune a set of dichotomizers already generated, for example using ECOC; in such a case, ECOC defines the target values for hidden units in an MLP, facilitating training. Simulation results on eight datasets indicate that compared with a linear one-per-class polychotomizer, pairwise linear dichotomizers and ECOC-based linear dichotomizers, this method generates more accurate classifiers. We also propose and test a method of incremental construction whereby the required number of dichotomizers is determined automatically as opposed to assumed a priori

Combinatorial Approach for Data Binarization

This paper addresses the problem of transforming arbitrary data into binary data. This is intended as preprocessing for a supervised classification task. As a binary mapping compresses the total information of the dataset, the goal here is to design such a mapping that maintains most of the information relevant to the classification problem. Most of the existing approaches to this problem are based on correlation or entropy measures between one individual binary variable and the partition into classes. On the contrary, the approach proposed here is based on a global study of the combinatorial property of a set of binary variable